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ABSTRACT 


Tills Technical Report addresses the general area of counter program 
testing. The test program for computer pro gr a m is. In many cases, 
different from that for equi p m en t Aw to the uniqueness of the ccespute: 
program configuration item. Thus, testing concepts for computer pro 
grass are not always widely understood. Ibis Technical Report attempts 
to clarify computer program testing requirements and procedures by con¬ 
sidering such topics as test requirements documents and test plans/ 
procedures/reports. The concepts of informal versus formal testing 
are introduced, which lead to the subjects of preliminary and formal 
qualification testing. Subprogram, functional, and computer program 
configuration item levels of testing are also explained, and these 
Ideas are in turn related to informal versus formal testing. 
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SECTION I 


INTRODUCTION 


1• General . Testing performed as an integral p~>rt of the acquisition 
process is governed by AFF. 80-14 and is audressec in this document as 
Development Test and Evaluation (DT4E), and Operational Test and Evalua¬ 
tion (OTAE). 'Hie purpose of the total test effort is to verify the perfor¬ 
mance requirements and compliance with specifie^t .ont of configuration 
items , subsystems, and the total, integrated 3jst<_a. 

2. Development Test and Evaluation . The DT4E effort is divided into 
the two areas of Cl/Subsystem Test'and System Test. Tne functions of 
eacn of the two areas of are given below. 

a. Cl/Subsystem Test . The Cl/Subsystea testing effcrt consists of 
the development testing and evaluation of the individual configuration 
items (CIs). subsystems and, in certain cases, the ccsnplet ? system. The 
Air Force actively participates in, evaluates, and controls the Cl/Sub- 
system testing; however, the tests are conducted predominantly by the 
contractor who ia under Air Force Systems Command (aFSC) c erection and 
control. 


(I) The overall objective of the Cl/Subsystem e’fort la the 
qualification of all CIs and subsystems or segments, thereoy preparing 
each element of the system for the subsequent System test orogram. The 
total objective is fulfilled by the following subordinate accomplish¬ 
ments : 


{a} Engineering test and evaluation necessary to the 
development of an acceptable design. 

(to) Preliminary Qualification Testing (FQ3) to confirm 
the functional integrity of mission critical functions. 

(c) Formal Qualification Testing (FQT) of each Cl, or 
group, to both environment and functional performance requirements. 

(d) Reliability test and analysis which confirms reli¬ 
ability goa?s and defines potential problems. 

(e) Integration of hardware, computer programs, and 
personnel subsystems. 

(f) Qualification of system segments or subsystems as 
specified 1 performance requirements. 

("’) Government control of the Cl/Subsysten test program Is 
established primarily through provisions of the contract and require¬ 
ments of th? specifications. Although aovemmert personn 1 will 
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normally not conduct any Cl/Su’o ay stem testing, it is intended that the 
SFO will designate government representatives, on site, to perform as 

monitors to determine the test progress, adherence to test procedures, * 

validity of collected data, and performance of the equipment and com¬ 
puter progr;uas. Normally, all Computer Program Configuration Item 
(CPCl) testing will be conducted as an integral part of tl c Cl/Sub- 
system effort. 

b. System Test . The System testing effort consists of testing arc. 
evaluation spanning the integration of subsystems into a complete system, 
and development tests of the completed system in as near an operational 
configuration and environment as practicable. System testing is an Air 
Force effort with contractor participation, under AFSC direction an! 
control, and with active operating and supporting command participation. 

Actual test operation and maintenance should be performed by military 
personnel who have received formal system training. 

(. ) The objective of System testing is the forral qualification 
of the system specification requirements. Specific objectives of the Sys¬ 
tem test effort are: 

(a) Demonstrate that the system can perfoia the mission 
a3 specified in Section 3 of the System Specification. 

(b) Verify results of Cl/Subsystem testing with Air Force 
crews in a live operational environment. 

(c) Qualify and/or demonstrate the performance of Cle 
which require full system operation for qualification. 

(d) Verify the adequacy and compatibility of the main¬ 
tenance and supply support concepts as developed. 

(e) Determine the safety characteristics > f the system, 
and procedures necessary to operate and maintain the system. 

(f) Provide sufficiently trained personnel for the operat¬ 
ing commands to assume operational evaluation tasks during the OT&E phase. 

(g) Verify required technical handbooks and manuals. 

3- Operational Test and graduation . The OT&E testing ef.'ort normally 
follows System testing and completes the testing phase of the Program 
Management Plan (HCP). OT&E tests will be conducted by tie appropriate 
operating command with technical support by Air Force Sys .ems Command 
and Air Force Logistics Command. 
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SECTION II 


TEST DOCUMENTATION 


1. Introduction . There are several types cf test doeumei nation «nich, 
collectively, form tne oasis for an effective test program. Broadly, 
these can be classed as: 

a. Test Requirements, which are Section k of each System (Type A) 
or Cl (Type 3 or C) Specification. 

b. Test Flans, which are usually the product of a validation phase 
and should reflect the overall planning for test aid evaluation of the 
system or a subsystem. 

c. Test Procedures, which are the detailed proceiur? 1 information 
for conducting each test delineated in the test p^ans (re: . "b' above). 

d. Tea* Reports, which summarize the results and analysis of each 
test conducted. 

2. Test Requirements Documents . Section 4, 'Quality Assurance,’ of each 
System (Type A) or Cl (Type B or C) Specification contains specified con¬ 
tractual re airements for testing of the respective systei or Cl. This 
specificati n section should depict h requirement to test each performance 
and design equireaent contained in Section “ of tne spec fication. 
Generally, Section 4 of a Type A specification will speci: y requirements 
for System Test, and Section 4 of a Type B or C speciflca ion will 
specify requirements for Cl/Subsystem Test. 

a. Section 4, S ys tem Specification . S .‘Ctioii , ' Ou 14 ty Assurance,’ 
contains the requirement:: for the system test program. Tnese requlre- 
ments must be relatable to performance/design requirements 3tated in 
Section 3- Therefore, 3«ction *» will normally be limited to system 
level test equirements, out will also include requirements for Cl/Sub- 
system engireering tests, qualifications, and reliability tests which 
can be accotgjlisred only at the system test location(s). The principal 
content of Section 4, however, is the specification of Sy. tern test 
requirements. 

u. lectio.. 4, CFCI Tart T Specif:.at Test reau rements are 

developed initially oy the contractor for incorporation i.ito Section 4 
(Quality Assurance) of the Fart I specification. This section should 
Identify test methods (to the level of detail necessary to clearly 
establish the scope and accuracy of the methods) to be employed In 
qualifying one CFCI agains all performance design requirements 
specified 1 i Section 3 of that Cl spec:ficati?.. In addition, it should 
identify requirements for gorvernaent-f urnishe i equipment and facilities 
to support the contractor*3 computer program test and evaluation, as well 
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as designate those performance characteristics to be demonstra .ed and/or 
verified during preliminary qualification tests and demonstrations, 
formal qualification testing, and System testing. 

(1) Requirements of Section 4 should be specified to ae level 
of detail which: 

(a) Designates verification requirements and methods for 
each performance/design requirement identified in Section 3. The methcl 
of verification to be specified nay include inspection, review of analyt¬ 
ical data, demonstration tests, and reviev of te. t data. 

(b) Clearly establishes the scope and accuracy of the test 

method. 

(2) Types of Cl/Subsystem tests which may be spec fled in 
Section 4 of a CPCI Specification are: 

(a) Computer Prograamiing Test and Evaluation which are 
tests conducted primarily to support the design and development process. 
They are listed in the specification only when they meet one of the 
following criteria: 

1 . They are intended to be the only sot .-ce of data to 
qualify specific requirements in Section 3. 

2. They must be accomplished as part ol an integrated 
teat program involving other systems/equipment/progr&rns. 

3 . They require the use of government- 1 irniahed test 
facilities or equipment. 

(b) Preliminary Qualification Tests, which ure formed tests 
oriented primarily towards verifying portions of the CPCI prior to forma}, 
qualification tests of the integrated CPCI. 

(c) Formal Qualification Tests, which are formal tests 
oriented primarily towards testing of the integrated CPCI, using oper¬ 
ationally configured equipment. 

( 3 ) The System test requirements to be identified in Section 4 
concern those performance/design requirements which canno* be verified 
until System testing. Emphasis should be placed on minimizing these 
types of requirements and, if possible, eliminating them altogether. 

This will ensure that the CPCI that ib Cl/Subsystem testeu has in fact 
attained a high degree of confidence. Also, chis will allow the System 
test program to proceed with its prime objective, testing of system 
level requirements. 

(4) An important point to remember is the differentiation 
between Section 4 and other testing documents. Section 1», by virtue of 
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being part of the Part I CPCI Specification, is a contractual document, 
and the contractor is required to conduct only those tests that are 
called out in that section. Any changes or deviations from these re¬ 
quirements (once baselined) must be approved by the System Program 
Office (SPO) through Engineering Change Proposal (ECP) actions. Thus, 
careful attention must be given to ensuring that the testing has been 
scoped properly, and the test requirements are complete and acceptable. 

3- Test Plans/Procedures/Reports . Figure 1 shows a typical test docu¬ 
mentation ''tree" for a large system. The tree is ccnstruetea from the 
list of approved test plan/procedure/report data item descriptions (DIDs) 
contained in AFSCM/AFLCM 310-1, Vol II. 

a. Syst e m Test Flan . The System Test Plan, DID T-101-1, is the 
"top document' for the test program which structures and unifies all 
subsequent test plans. Its purpose is to provide an overall outline 
of the total system test program to include planning fact ra, objectives, 
and scope of all phases of the test program. The system ontractor pre¬ 
pares the System Test Flan (DID T-101-1), normally for de lvery in the 
Conceptual or early Validation Phase. This plan will con ain basic test 
planning information for all phases of the test program a~.d cover the 
life cycle of the system through the end of the Full Seal" Development 
Phase. The scope and level of detail of the information as sufficiently 
broad and comprehensive to provide basic test inputs to t.-e Program 
Management Plan (PKF) and sufficiently detailed to provide the basis 
for preparing all subsequent test plans—particularly Cl/.ubsystem 
(DID T-102-i and DID T-103-1) and System (DID T-106-2) plans. 

(1) The System Test Plan (DID T-101-1) is based upon the test 
concepts and requirements contained in the Program Management Plan (PMP), 
the System Specification, and related system engineering ocumentation. 

It covers all aspects of the system, including hardware, facilities, 
computer programs, personnel, and procedural data with re-pect to such 
consideration as the following: 

(a) Organizational responsibilities for testing. 

(b) Basic system test concepts and objectives for Cl/Sub- 
systea. System, Implementation, and Acceptance tests. 

(c) Overall system test operations, including test control 
requirements and test support requirements. 

(d) Test evaluation requirements. 

(e) Test reporting requirements 

(f) Overall test schedules. 









(2) A primary function of the System Test Plan (DID T-101-l) 

Is to guide the contractor's planning, analysis, and system engineer¬ 
ing activities related to the test areas. During the Validation Phase 
the contractors expand basic planning and guidance information contained 
in the System Test Plan (DID T-101-l) and, in effect, replace it with 
the following documents: 

(a) Cl/Subsystem Test Plan (DID T-102-1 and T-103-1). 

(b) Inputs to the System Test Flan (DID T-lOfe-2). 

(c) Inputs to the Implementation Test Plan (multisite 
systems; DID T-.114-1). 

(3) Special problems to be considered concerning or uputer pro¬ 
grams may include provisions for computing equipment required for con¬ 
tractor development and testing of computer programs. The Cl-Subastea 
test cycle for computer program Cla will normally begin early during 
the Full Scale Development Phase; it may have to be initiated using a 
prototype computer and peripheral equipment, or in same cnees by slam 
latlon on existing computers. As development of the CPCI progresses, 
testing will encompass progressively expanded groups of routines, and 
will require additional items of computing equipment for realistic test¬ 
ing of functions. Preliminary qualification testa/demonstration will 
normally occur at the contractor's plant, or other locations of available 
equipment. Formal qualification of the computer program Cl may require 
scheduled time at the System test site for testing those computer pro¬ 
gram functions that depend upon the operationally configured system/ 
equipment and for which simulation is not adequate. For a multisite 
system which requires adaptation of the operational computer program 

Cl at each site location, the expected sequence of events at each fol¬ 
low-on site will be installation and checkout of equipment at the site 
facility, installation testing, adaptation, installation and checkout 
of computer programs, and implementation testing (See Figure 2). 

b. Cl/Subsystem Test Flan/Procedures/Reports (Equipment) . The Cl/ 
Subsystem Test Plan/Procedures data item, DID T-102-1, is the overall 
planning document for the Cl/Subsystem test program. It is subordinate 
(in the test planning tree) only to the System Test Plan (DID T-101-l). 
this document should provide complete planning information on each Cl/ 
Subsystem test specified in each of the Cl specifications on contract— 
l.e.. System Specification, Cl Specification, Critical Component Speci¬ 
fication, Military Specification, and other contractual documents that 
include Cl/Subsystem test methods and success criteria. Rote that test 
planning and procedures Information for Cl/Subsystem testing of computer 
programs should not be included in this document. In the case where the 
system Involves computer programs, an additional test data item is used 
to detail the planning and procedural information for testing of the 
computer programs. Ibis document is Cl/Subsystem Test Plan/Procedures 
(Computer Program), DID T-103-1, which should be referenced in DID T-102-1. 



This data item la normally obtained in the Validation Phase as a com¬ 
plete plan and individual test information sheets are then updated In 
the Full Scale Development phase prior to each test. Test reports are 
obtained tinder DID T-118-1. 

c. Cl/Subaygtea Test Plans/Procedures/Report (Computer Programs). 

The computer program CI/Subsystem Test Plan/Procedures, DID T-103-1,is 
a subordinate and supplementary document to the overall Cl/Subsystea 
Test Plan/Procedure, DID T-102-1. When the system involves computer 
programs and equipments, DID T-103-1 should be placed on contract 
along with DID T-102-1; DID T-102-1 will apply to the Cl/Subaystea test¬ 
ing of all equipments and reference DID T-103-1 which will cover the Cl/ 
Subsystem testing for the computer programs. 

(l) The Cl/Subsystem Test Plan for computer programs is a 
contractor-prepared document which establishes criteria, general methods, 
responsibilities, and overall planning for Cl/Subsystem testing of a 
CPCI. Normally, the test planning information is obtained in the Vali¬ 
dation Phase as a complete plan applicable to all compute 1 ' programs of 
the system. Generally, only one plan (and one volume) will be prepared 
for a single system or contract—this allows for the presentation of an 
integrated test plan which will apply to all computer programs for a 
particular system. Exception to this guidance can be obtained for the 
individual system. Sections of the plan which apply to computer pro¬ 
grams for subsystems for which inadequate information exiBts at time of 
writing of *he plan, can be updated at a later date. When this is the 
case, the section should be included nonetheless with the remark, "to 
be completed later." 

(a) The plan should contain detailed information concern¬ 
ing the implementation of preliminary and formal qualification testa, 
along such lines as the following: 

1. Locations at which the tests will be conducted, 
and schedules relative to milestones in the overall acquisition schedule. 

2. General methods for preparation of input data— 
l.e., simulation and/or generation vehlclea to be used. 

3 . General procedures for test conduct, and respon¬ 
sibilities for test direction, operation, and observation. 

4. General procedures for analysis of test results. 

5 • Requirements for other computer programs, equip¬ 
ment, and facilities. 

6. Personnel requirements, including numbers, re¬ 
sponsibilities , and particular knowledge and skills required. 
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(b) The plan Bay also set forth the requirements and 
procedures for controlling and documenting the Cl/Subsystem test 
program, including procedures for preparing, reviewing and revising 
documentation of specific test procedures; requirements and pro¬ 
cedures for preparing and reviewing reports of individual quailfl- 
catlon tests; sumarles of the Cl/Subsystem test program or phases 
thereof; and other reports related to the Cl/Subsysteo test activity. 

(2) The Cl/Subsystem Test Procedures are produced 'ey thu con¬ 
tractor during the Full Scale Development Phase. 

(a) For each individual Cl/Subsystem Qualification Test 
(PQT or FQT) for a CPCI, the contractor will prepare & Cl/Subsystem 
Test Procedure in accordance with Section 2 of DID T-103-I. These 
procedures are prepared incrementally during the Full Scale Development 
Phase and submitted prior to the test date of the PQT or FQT to which 
they apply. Information included in a test procedure includes the 
following: 

1. Loc&tion and schedule of the test, oriefings, de¬ 
briefings, and any associated data reduction/analysis. 

2. References to applicable test plan, specifications, 
manuals and handbooks. 

2- Detailed objectives of the test. 

h. Requirements and responsibilities for console 
operators, -est directors, technical consultants, data analysts, or 
other essential teat personnel. 

5 . Requirements for other computer programs (other 
than the CPCI being tested) or equipment. 

6 . Test operating procedures to specify how to 
initiate the computer program operation, maintain the computer program 
operation, and terminate and/or restart the computer program operation. 

2* A detailed test description for each test (or 
portion of a test) to be performed. This description should Include 
detailed Information on test Inputs, outputs, events, expected results, 
reactions to be verified, and methods of verification. 

8 . Requirements and procedures for recording, re¬ 
duction and analysis of test data. 

(3) The Cl/Subsysteo Test Reports for computer program tests 
are obtained under DID T-118-1. These are contractor prepared docu¬ 
ments, compiled incrementally during the Full Scale Development Ffcsse. 
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Generally, one test report (in accordance with BO T-118-1) is prepared 
for each PQT or FQT conducted. 

d. System Test Plan/Procedure /Report . BO T-106-2 is the basic 
document which provides the overall integrated outline of the System 
Test Program. It includes all planning factors, scope, detailed test 
objectives, identification of test areas, responsibilities of partici¬ 
pating agencies, and associated information necessary to implement the 
minimum acceptable performance requirements of the Program Management 
Directive (FWD) and/or the System Specification - Normally, the con¬ 
tractor will prepare the System Test Flan and Procedures ;DO T-106-2) 
with the Intent being to establish a test and evaluation program to 
ensure that the system/equipment/computer programs meet the minimum 
acceptable performance requirements of the FMD and System/CI Specifi¬ 
cations in as realistic and complete an operational environment aa 
practicable. Test Reports are obtained under BID T-120-2 

e. Installation Testing . Installation Testing, normally conducted 
after completion of each installation, includes ?reshakedown Teats, 
which are performed as a combination alignment check and test to assure 
that the installation is properly completed; Shakedown Tests, which are 
performed to assure that all detected marginal parts and material have 
been eliminated and that the installation is ready for op'rational 
tests; and "oeration*... Tests, which are performed to demonstrate that 
the equipment is properly nstailed and is capable of performing its 
operational mission up to a specified interface with other portions of 
the subsystem/system. 

(1) The Installation and Checkout Flan (I&C), DIB T-112-1, is 
initially prepared by the contractor during the Validation Phase and 
later expar Ved to reflect the system engineering and detail design 
activity oi the Pull Scale Development Phase. Procedures are prepared 
to implement the plan, based upon assembly levels of components and 
consideration of all interfaces. In general, individual items are in¬ 
stalled, physically and functionally checked out, then *a -ed to other 
subsystems. This graduated process is applied successively until the 
complete system is ready for system tests. 

(2) IBC plans are reviewed and approved by the SPO and the 
procedures selectively reviewed and approved. 

(3' Adaptation, installation, and checkout (AISC) of the com¬ 
puter program will normally occur after installation and checkout of 
equipment and facilities. Normally, the work will be accomplished by 
the contractor in acc'roar.ee with the approved plans and procedures 
obtained under DIB T-112-... At the System Test Site, AI6£ of the com¬ 
puter prog: am may be followed by Form ad Qualification Testing prior to 
initiation of the System Test. Based upon the experience gained at the 
System Tea*. Site, the plan may oe modified and updated for use at 

each follow-on site. AI&T of computer programs at follow-on sites will 
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include the sane basic activities as for AI iC at the System test site 
and will occur in the sane general sequence, i .e., following ISC of the 
equipment/facllltles, but prior to the start of Implement*'ion Tests 
(see Figure 2). For computer programs, appendices m&j be used to cover 
unique adaptation features for each site. 

(a) The computer prograa contractor aust identify require¬ 
ments for AISC of the cosqwter prograa at the Systea test site, and for 
multisite systems, must also include requirement- for A18C at juibseqcent 
site installations. Documentation simile. Include a detailed description 
of all aspects of the adaptation, installation and cneckout activities. 
Detailed schedules should be provided and all support ret ilrenects 
clearly identified, including required computer time, requirements for 
other systea equipments such as ccosunlcation, display consoles, etc., 
and requirements for trained personnel. Detailed informa.ion should 
also be provided concerning the training of personnel for the field 
locations, scheduled movement of field personnel, local t ansportation, 
office space and facilities, and living quarters at remet.ly located 
sites. Procedures should be prepared indicating in detail the exact 
manner in which the plan will be implemented. 

NOTE: M Adaptation’’ refers to the process of inserting into the computer 
program the coded data which are appropriate to the geography or other 
characteris ics of the given site. "Installation 1 refers to insertion 
of the codet instruction/data into the computing equipmen ; in contrast 
with equipment installation, computer program installatio typically 
involves relatively insignificant effort or time. The ma.or task of 
AISC is the checkout activity, which may in some cases in olve assembly 
of new elements, compiling, and extensive debugging, as w 11 as person¬ 
nel training and preliminary rehearsals of FQT procedures. 

f. Implementation Testing . Implementation Testing is a concept which 
has been derlned to meet the specific requirements of multiBite (i.e., 
ground installation, airborne vehicles, space platforms, or tactical 
emplacement.-.) electronic systems. The concept is based cu experience 
which has shown that additional testing beyond Installation Testing may 
be required at successive operating sites to Insure that these sites, 
individually and collectively, function as parts of a system in accor¬ 
dance with the requirements of the systea performance specificaticc. 

These tests encompass real-time functional testing perfor led at the 
system level with the purpose of exposing faults and providing a demon¬ 
stration to the user that the installed site is ready fox operational 
use. 


(1' The relation of the System Testing and Implementation Test¬ 
ing should be one of decreasing complexity and sophistication. To the 
extent that system performance requirements set forth in the system 
specificat on have been demonstrated by Systea testing, a complete re¬ 
petition o^' the effort for each follow-on system is not warranted. 
Rather, sp- cific functional performance measures which adequately define 


11 



















system response are Identified at an early point In time, and the actual 
rallies for such performance measures (figures of merit) are determined 
during System testing. These figures of merit are used dnr.-jg Imple¬ 
mentation Testing as the criteria toy which to gauge functl .ai adequacy 
of the system. In general. Implementation Test - should n^t attest tc, 
duplicate the comprehensive testing accomplished during System testing. 

(2) To insure operational validity, the user as well as the 
system designer must participate in Implementation Test planning. The 
user is interested in a test that will uncover technical Incmpatlbllitles 
or functional and operational deficiencies. Thus, the nacner in which 
the system is to be employed Is a vital factor in test planning. 

(3) The Implementation Test Plan and Procedures arc contractor- 
prepared documents obtained under DID T-ll^-l. The Implementation Test 
Plan defines the overall scope of the impictent&tior. test;, the objec¬ 
tives, methods, and support requirements for the conduct f this phase 
of testing. The Implementation Test Procedures (or.e se\ :i p. oceaures 
for each test conducted) outline a step-by-step set of instructions and 
criteria for each test specified in the plan. Test result a are reported 
under DID T-119-1. 

(h) The computer program contractor(s) prepare 1 puts to the 
Implementation Test Plan/Procedures aaich the sane as for .he System 
Teat Plan/Procedures. Since impierentation testing is at tne system 
level, emphasis will be placed on those aspects of the computer program 
performance which depend upon complex interactions with other CrCIs, 
personnel, equipment, and data links during operation of the entire 
system. 

(a) Special emphasis will be given tc those portions of 
the CPCIs (generally the operational CPCI) which vary from site-to-site, 
i.e., environmental data values, special adaptation parameters, unique- 
to-site interfaces, and site-peculiar progrem x-outii.es. 

(b) Inputs tc the Implementation Test Plan relative to 
computer programs should be similar to the inputs to the System Test 
Plan (DU) T-106-2). In general, they should emphasize those aspects of 
system performance which are closely associated with or dependent upon 
performance of the CFCl(s). Frequently, the inputs may represent a 
selected subset of the System Test Plan (DID T-lOo-2) inputs. 

Section 4, CPC I Paxt IT Specification , Koraally, Section h of a Cl 
Part II Specification will contain quality assurance provisions and 
acceptance test requirements for follow-or. production items. However, 
unlike equipment items, there is no foilov-on production process for ft 
computer program item; thus, the concept of acceptance tests for pro¬ 
duction items will not apply to a CPCI. Instead, Section ^ of a CFCI 
Part II Specification should contain two subsections: 


a. Test Plan/Procedure Cross Reference Index . This subsection 
should contain a cross-reference diagram depicting each function (as 
delineated in the CPCI Fart I Specification) and relating thesu func¬ 
tions to the corresponding test plan/ procedures that were used to 
qualify the individual require»ent8 of the CPCI. 

b. Other Quality Assurance Previsions . This subsection .ou_d 
reference and/or specify the teat/verification rectireaects , methods, 
and procedures which apply to preparation and duplication oi the con- 
puter program (i.e., tapes, card decks, etc.). 




SECTION III 


COMPUTER PROGRAM Cl/SUBSYSTSM TEETDG 


1. Introduction . 'Hie Cl/Subsystem testing of CPCIs serves a faurfctd 
purpose: 

a. To establish each CPCI as a qualified end item suitable for entry 
into the System Test Program. This qualification is accomplished by 
verifying the performance and design requirements of the CFCI Peart I 
Specification. 

b. To furnish the procuring agency with the proper visibility re¬ 
quired for effective management of th» systea- This i3 accomplished 
through judicious scheduling of PiTs, thus establishing milestones which 
will provide insight into the progress of the CPCI design and develop¬ 
ment. If problems are encountered in the computer program area, early 
detection can be made and more effective management and engineering can 
be focused on these potential problem areas. 

c. To serve as a standard or "straw man" about which the contractor 
can develop bis internal verification procedures. 

d. To develop a comprenensive test (called F3T) which can be utilised 
after FCA as an .-ingcing test tool/procedure for retesting the CPCI. FQT 
is not only used for formal verification prior to FCA but will be con¬ 
tinuously updated and used tc reteet the CPCI whenever changes are made 
to the program. 

2. Types of Cl/Subsystea CFCI Testing . Cl/Subsystem testing of CPCIs 
can be broadly divided into two types--infonaal testing and formal test¬ 
ing. The basic difference between there typer fiter* f:-na the documen¬ 
tation requirements; informal testing utiliz ?<■ -he c on tractor : s internal 
test documentation, controls nr.d r-rot»bu5 e.<-; forma. 1 testing is conducted 
in accordance with Air Ferre approved test plans and procedures. 

a. Inform a l Testlr .r. CT'S'lhp.-t'-a ir.f~.-nai testing (referred to as 
Ccfflaier Program T: _ t or.i Evaluation. ' will usually fora the buljt 

of the contractor's f I/c.u'csyrtem test effort. It is designed to be the 
contractor's 'in-house' testing and requires no government approved test 
plans/nrrvreiurc-s . ‘teners ily, the entire i . formal tea*, effort will be 
documented ir.torr.nul.y by me ~retractor, .tn J such info-antic® will be 
made available t~ the pro-- wring agency ~niy rr, oe-ard. Informal testing 
begins when the first cuonrocram is ceded r.rd cortlruas throughout the 
Full Scale Development Phase. 

(l) Informal testing retai l’d, rr three ie ,r elr for a *t*rp-by- 
otep validation of the CPCI. 
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(a) Subprogram level (parameter testa) 

(b) Functional level (assembly tests) 

(c) CPCI level (assembly tests) 

(2) Parameter tests sire tests of the information processing 
logic of the individual CFCs. Sets of inputs are prepare^ vhich caui«e 
all the logic paths to be exercised. The resultant outputs are checked 
against hand calculations which show whether the CPC has been codec 
correctly. 

(3) Assembly ~<_ats are tests of groups of CFCs woich perform 
a function of the CFCI, or in later stages, many or all o:' the func¬ 
tions of the CPCI. Interfaces between the CFCs as well as overall 
information processing are tested. Simulated inputs to t e CPCI are 
used, and the CPCI is operated in real-time or simulated eal-time 
mode. 

(4) The informal test activities are conducted solely for the 
purpose of providing information required for the developmental process. 
As such, they do not require recognition by the procuring agency and 
would not ntrmally be identified in the Cl/Subsystea Test Plan. The 
planning anc conduct of informal testing is carried out ir. accordance 
with the contractor's internal management procedures, and within the 
time constraints impose- oy formally scheduled reviews an : qualification 
testing. Each CPC or subprogram must pass, through a seri '8 of stages 
and iterations, consisting of such operations as desk checking; elimi¬ 
nation of illegal instructions, parameter tests, functional testing with 
controlled data inputs, assembly with additional ccnponer 8 of the CFCI, 
assembly testing, performance testing with simulated inpt ts and, finally, 
performance testing in the system under conditions of live operations. 

(5) The Cl/Svosysv-em Test Plan will not generally Include plan¬ 
ning lnforn.ition for contractor testing which is conducted as an integral 
part of the design ana development process. However, the plan will 
typically octal! the contractor's requirements for government facilitiep 
and other support requirea for conducting the tests. These will include 
requirements for computing and peripheral equipment which must be fur¬ 
nished or otherwise mane available for the contractor'a use on an ap¬ 
propriate schedule during she Tull Scale Development Phase. Where 
formal qualification of tne C?Cl is scheduled to occur at the System 
test site, it will normally be preceded by a phase of contractor test 
and evaluation associated with adaptation, installation, and checkout 

of the CFCI(s) at the sit': The Cl/Subsystem Test Plan should include 
(or reference other planning documents which include) requirements for 
scheduled use of the facility, system equipment, other computer program 
CIs, personnel, and other items needed to support the conduct of these 
contractor tests. 





b. Formal Testing . Cl/Subsystem formal testing is that portion of 
Cl/Subsystem testing which la conducted in accordance with Air Force 
approved test plans/procedures. Thus, the Air Force has explicit con¬ 
trol of the type, number, and frequency of tests to be performed. 

(l) The Cl/Subsyste* formal test effort is divided jot.c two 
distinct types of testing—Preliminary Qualification Testing (PQl) and 
Formal Qualification Testing (7QT). PQT is designed to be an incre¬ 
mental process which will provide the procuring agency proper visibility 
and control of the computer program development during the time period 
between the Critical Design Review (CDS) and FQT. FQi is designed to be 
a complete and comprehensive test of the CPCI in a 'one step" faahlon 
Just prior to FCA. 

(a) Preliminary Qualification Testing (PQT) it composed 
of function level tests (assembly tests) which are conduc ed in accor¬ 
dance with Air Force approved test plans/procedures procured under 
DID T-103-1. FQT is to be an Incremental process vhicn occurs between 
CDR and FQT of a CPCI's development process. For each function (of the 
Fart.I CPCI Specification) which is designated for testing during FQT, 
a separate test procedure i3 written and a formal test conducted. Mott 
that, generally, not all functions of a CPCI are tested dnrir^ FQT since 
experience naa proven that it is both too costly and too time-consuming• 
Instead, omy designated functions of the CPCI are tested during PQT; 
thus, the problem becomes one of the selection of those functions which 
should undergo PQT- Selection of these functions should se baaed on 
the following: 


1. A PQT should be conducted for e:.-b (’unction that 
is "time-critical" to the development of the CPCI, subsystem, or system. 
For exsu^ile, the compiler function of a utility CPCI may be designated 
"time-critical" if the development of the remaining computer programs 
depend on the timely development of a compiler to be used for compiling 
the other programs. The development of the executive function of am 
operational CPCI may also be designated tine-critical" since the 
orderly progression from parameter testing to assembly testing of the 
operational CPCI would require the timely development of the executive 
function. 


2. A PQT should be conducted for each fuucticn t.iat 
is "performance-critical to the development of the CPCI, subsystem, or 
system. For example, the executive function could be designated "per¬ 
formance-critical" as well as "time-critical due to the utilization of 
various exotic scheduling techniques. As another example, the tracking 
function of a CPCI foi ar. or.-line radar system might be designated "per¬ 
formance-critical" due \v, -he utilization of new tracking, smoothing, 
and filtering techniques. 

(b) Preliminary Qualification Tests (PQTs) should normally 
be conducted at a contractor's development facility (i.e., contractor's 
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plant or other location of available equipment), typically using con¬ 
trolled input8 specifically prepared for the test/demonstration purpose. 
The Test Plan should outline the sequence of individual and/or assembled 
CPC tests, identify the CFCI performance/design requirements to be veri¬ 
fied at each PQT, and identify special siaulatiou/recording equipment 
or other support requirements for the PQT program. Each Test Procedures 
document should be completed by the contractor and submitted to the 
monitoring agency sufficiently in advance of the scheduled test session 
(i.e., two to four weeks) to permit revie j ana analysis f the procedures 
prior to witnessing the testing operations. The format nd content of a 
typical Test Procedures document are specified in DID T-103-1. 

(c) Formal Qualification Testing (FQT), unlike informal 
testing and PQT, is not an incremental process. FQT is designed to be 
an integrated and comprehensive functional test of the CPCI as a whole, 
and usually is conducted in one continuous time period ju°t prior to 
FCA. Each function of the CPCI (as delineated in the CPCI Part I Speci¬ 
fication) is tested during FQT, regardless of the amount of Informal 
testing and PQT conducted previously. Since PQT is conducted during 
the on-goin? design process, the PQT test procedures became obsolete 
within a fe. weeks after the PQT tests. In contrast, since FQT is con¬ 
ducted after the design process culminates (just prior to FCA), the FQT 
test procedures can be maintained end updated and used throughout the 
remainder of the Full Scale Development and Deployment Phases. Thus, 
each time a change is implemented in a CPCI, the FQT pre 'dures can be 
used as a test tool to retest the CPCI and ensure that it still functions 
in accordance with the functional requirements set forth in the Part I 
Specification. 


1. For “he less complex CFCIs, or for those (i.e., 
utility) which are relar.i .eiy insensitive to the system operations, 
formal qualification will usually be conducted at the contractor's 
development facility. However, for a large operational CrGI, the sheer 
complexity of the performance requirements may dictate that FQT can only 
be conducted in the context of the operationally-configured system, in¬ 
cluding personnel and conraunications. Hence, for these cases, FQT may 
often not to conducted at the contractor's plant (development facility), 
but may require U3e of the System test site prior to the Wglnnlng of 
System testing. 


2. Ir. t-e event that a cocplex operational CPCI is 
scheduled for formal qualification at the System test site, scheduling 
of the FQT will normally be prior to the start of formal System test¬ 
ing, but following a period of contractor adaptation, installation, and 
checkout of the CPCI ir. the previously installed and tested operational 
equipment/facilities. 

3 - In either case, FQT will involve the use of sim¬ 
ulated and controller inputs which can be designed to cover the expected 
ranges of ’-ariables, system operational modes and conditions. Including 
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capacities and limits. Pull qualification of certain functions may 
depend upon subsequent System testing with live inputs and communi¬ 
cations . 

4. The procedures for Formal Qualification Tests 
(ref DID T-103-l) will be in the same format as the PQT procedures 
prepared during design and development. However, these procedures will 
emphasize verifying that all functional requirements are met, baaed cn 
the demonstrated performance using the complete CFCI. Since the FQ'f 
procedures require approval by the monitoring agency, they should be 
submitted initially in preliminary form. Review and analysis may result 
in proposed changes which must be resolved and reflected in revised 
preparations for the testing activity. Hence, the preli m ina r y document 
should be completed in sufficient time (i.e., three months prior to 
scheduled PQT) for xnls potentially lengthy process to be accomplished. 

3 . Levels of Cl/Subsystem CFCI Testing . As mentioned earlier, testing 
of a CPCI is required at throe levels: subprogram level, functional 
level, aad CFCI level. Regardless of the type of testing to be per¬ 
formed (i.e., informal versus formal testing), these three levels apply 
equally to each. This relationship is portrayed in Figure 3* 

a. Subprogram Testing . Subprogram testing, sometimes called para¬ 
meter testing, is directed toward ensuring that each subprogram (CPC or 
lesser entity) interprets its inputs correctly, successfully performs 
all tasks defined in the subprogram coding specifications, and adheres 
to prescribed coding conventions and standards. Subprogram testing is 
the most detailed and basic testing that is performed upon computer 
programs. Since concentration is on testing basic, easily managed units 
of code, the detection, isolation and correction of errors that are 
exceedingly difficult to detect, isolate and correct in lauer phases of 
testi.g is greatly facilitated. Therefore, detailed attention must be 
given to the specification of rigid test requirements sc that errors 
are not overlooked that will later cause unnecessary testing and analysis. 
Sach subprogram should be specifically tested for arithmetic and logical 
accuracy and limitations, usually without acquiring communication with 
related subprograms or external equipment. Subprogram testing c emine nces 
following the development of a set of coded instructions that have been 
aubjecued to detailed visual verification (code checking) by the respon¬ 
sible programmers, said automatic code analysis by the applicable compiler 
and/or assembler to eliminate errors in keypunching, formatting of code, 
etc. In addition, a thorough technical review of the subprogram should 
be conducted by contractor personnel to ensure that violations of cod¬ 
ing conventions and rtandards have been corrected prior to on-computer 
testing. The suborr re . is then operated with a range of data that 
forces use of ail decision points and processing paths. The test re¬ 
sults are analyzed to detemine whether the derived results are con¬ 
sistent with the input data, the results expected from the selected 
input parameters, and tne operation of the subprogram as defined in cod¬ 
ing specifications. 
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b. Functional Area Testing . Functional area testing, sometimes 
referred to as string or assembly testing, is directed to. ard ensuring 
that selected sets of functionally related subprograms interpret all 
Inputs correctly, successfully perform all processing tasks specified 
in performance specifications, end generate output data that satisfies 
the Input requirements of other Interfacing functional areas or equip¬ 
ments. Functional area testing Is an extension of subprogram testing 
with emphasis being placed on inter-program communications and pro¬ 
cessing of data within a defined grouping of subprograms or components. 
Subprogram testing, on the other hand, concentrates cn intra-program 
communications and data processing. Examples of functional areas are 
radar inputs processing and correlation, data link inputs processing, 
and weapons guidance. Each functional area Is composed of one or more 
subprograms that must accept as inputs other functional area outputs, 
or Inputs received via the operating hardware, and process them in 
accordance with functional performance specifications. It must also 
prepare data for use by other sets of functionally relatea subprograms 
as specified in detailed coding specifications. In some cases, (l.e., 
tape read), functional area testing may be the first phase of testing 
performed. Ibis will most often occur when subprogram testing requires 
the excessively expensive simulation of interfacing hardware. 

(l) Functional area testing is usually the first point at which 
interactions with a control subprogram are examined. Ideally, the ex¬ 
ecutive or control function is tne first to enter the functional area 
test phase, since the other functions can be more conveniently tested 
with it. In actual practice, however, the control function may not be 
available or far enough along in development for this purpose. In this 
event, other methods for control may be utilized. Although many func¬ 
tional areas are usually being tested at the same time, tne scope of 
testing gradually increases through higher and higher levels of assembly 
as function*; become verified until the computer program is ready to 
undergo CPC I testing. In practice then, functional area testing begins 
as "high-level” subprogram testing and phases out as "low-level” CPCI 
testing. 


(2) Throughout this phase of testing, the performance of the 
computer program Is checked against that specified in performance re¬ 
quirements decisions and limits that result from an lnteiface between 
subprograms. Therefore, conditions that are of consequence to only one 
subprogram, such as proper exits in subroutine decision paths, are 
better tested in the subprogram test phase when the problems of defining 
and establishing the proper test situations are not so complex. In ad¬ 
dition to differences in levels of concentration, functional area test¬ 
ing examines interactions be .een all functionally related components 
while CFCI testing concentrates on interactions between functions. Also, 
internally stored inputs t -.t are provided to the subprograms during 
the functional area test pnase usually result from the Insertion of test 
data by a test Input tool. CPCI testing, on the other hand, uses Cl 
level inputs that are provided either by other CPCIs, or by system In¬ 
put data generators. 
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c. CPCI Testing . CPCI testing is directed toward ensuring that the 
total package of computer program components that comprise the CPCI 
correctly interprets all input message typeo and values, accepting 
those that are legal and properly disposing of all others; properly 
processes all Internally stored data; and correctly formats, arranges, 
and outputs all required system data. Thus, CPCI testing verifies that 
the CFCs operating as the CPCI fulfill the requirements of performance 
specifications. Whereas subprogram testing concentrates on verifying 
the correct operation of Individual computer program components, and 
functional area testing emphasizes inter-program coosunicatlons within 
separable functional areas, CPCI testing concentrates on verifying that 
the cosplete set of ccoputer program functional areas correctly Inter¬ 
acts with each other. Therefore, CPCI testing must be accomplished 
after functional area testing and prior to, or in parallel with, test¬ 
ing with other CIS. CPCI testing can be conducted at either the con¬ 
tractor's in-plant test facility or an installation facility (either 
operationally employed or a nonoperational test site). During the 
Initial stages of CPCI testing, the bulk of the testing la conducted 
in-house using simulated inputs. This provides a high degree of con¬ 
fidence prior to the release of a Cl to a site location. During the 
later stages of CPCI testing, emphasis shifts from the testing of func¬ 
tional coBBBunication to verification of overall CPCI performance in an 
operational (or very nearly so) environment. 




